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Influence of Subject Characteristics, Incentives and Multiple 

Mailouts on Survey Responses 



Abstract 

This inquiry was conducted to examine potential influences 
on perceptions of survey subjects regarding characteristics of 
the subjects, incentives to enhance mail returns and multiple 
requests for information. A mail survey sent to 297 former 
students produced responses from 48.3 percent of the sample. 
Analyses of returns yielded the following results: 



1. Sampling former students across level of teaching experience 
and year of graduation did not result in a response bias to an 
instrument seeking perceptions on topics of general 
professional interest. 

2. Using incentives to increase the number of responses to a 
survey did not introduce a response bias, although using a 
small monetary incentive {25 cents) was nonproductive in this 
inquiry. 

3. Using multiple mailouts to increase the number of responses 
did not result in differences in response patterns across 
mailout requests. 



ERIC 



3 



influence of . • . 2 



A variety of techniques for enhancing survey returns can be 
found in the literature of marketing, sociology, political 
science and education. These techniques include pre-contacts 
(advanced notification of the upcoming survey), multiple and 
intensive follow-up correspondence to non-respondents, hand 
stamped postage and personalized cover letters, and the use of 
incentiv/es both monetary and nonmonetary (Armstrong, 1 975 ; 
Dillman, 1978; Furse, Stewart & Rados, 1931; Heberlein & 
Baumgarliner, 1978 ; Huck & Gleason, 1974 ; Tollefson, Tracy & 
Kaiser, 1984; Zusman & Duby, 1984), Yet Jackson and Schuyler 
(1984) have reported mixed results through their application of 
techniques thought to enhance recurns. To illustrate, they 
reported: fewer responses were remitted when "cute" reminders 
were sent rather than businesslike requests and when instruments 
were mailed in April rather than F-bruary; no dj^lferejice in 
responses occurred when expensive stationary was used compared to 
photo copied letters, when items were mailed bulk rate rather 
than first class, or when surveys were mailed to arrive on 
Wednesday rather than Saturday; more responses occurred when 
reminders which included a new instrument were sent. 

The prepayment of monetary incentives is thought to be a 
powerful technique for motivating mail survey participation. 
Armstrong (1975) and Linsky (1975) reported monetary incentives 
have consistently resulted in higher response ratios (ratio of 
individuals surveyed who complete and return instrument divided 
by total number of subjects surveyed). Additionally, the most 
cost-effective amount of money to offer is considered to be a 
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quarter, yet greater response ratios have resulted from one 
dollar incentives. Further, monetary in-^entives have been found 
to be most effective when enclosed with the first mailing rather 
than promised upon receipt of the completed questionnaire 
(Armstrong, 1975), Firially, although there is concer^i that the 
use of monetary incentives introduces bias, there is no evidence 
to support this potential problem (Nederhof, 1983; Zusman & Duby, 
1984) . 

The present study which incorporates elements from survey 
research reported previously has attempted to extend this 
literature through examining potential influences on perceptions 
of subjects regarding ideographic characteristics of subjects, 
incentives to enhance returns, and multiple requests for 
information. 

While the preceding description delineates the purpose of 
the study, research questions were phrased to guide the inquiry 
process. These questions include: 

1. Does the date of graduafion and level of work and 
experience influence the perceptions of respondents? 

2. Do incentives influence the response ratio and the 
perceptions of respondents? 

3. Do perceptions of subjects who responded to an initial 
request differ from perceptions of subjects who 
responded to a second request for inf orrr.at ion? 

He t hods 

This survey was conducted during the 1985 spring semester. 
Typically, information gathering from former students is an 
iterative process within the College of Education occurring in 
conjunction with program development. Due to the high level of 
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activity in program development, five surveys have preceded this 
mail survey during the past decade that is, 1974, 1977, 1978, 
1982 and 1983. The 1978 and 1982 efforts sought information on 
graduate curricula while the remaining three surveys addressed 
undergraduate curricula . 
Subj ects 

A random sample stratified by year of graduation, was 
selected yielding 297 subjects. This sample represents 21.6% of 
the baccalaureate degree graduates (N=1375) from the Department 
of Educational Curriculum and Instruction at a large land grant 
university in the Southwest during a five year period (riay 1980 
through December 1984). The percent of male candidates during 
this period was 4.1%, and the percent of minority candidates was 
3.2%. The corresponding percentages of malp and minority 
candidates in the returns were 9% and 5.4%, respectively. 

The number of subjects selected for this inquiry was based 
on the assumption of a response ratio of .5 yielding 148 
completed instruments (297 * . 5). A sample of this size would 
provide the statistical power of .56 assuming an alpha level of 
.05 and an effect size of .20 for the various statistical tests 
conducted in this investigation (Cohen, 1977 . 
Instrumentation 

One instrument requesting the subject^s name, current 
address, year of graduation, current occupational status, and 
containing 43 Likert type items associated with pedagogical 
knowledge end skills was developed for this inquiry. FolJowing 
the:;e items, three additional questions were listed which sought 
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yes/no responses regarding a departmental newsletter. The 
instrument concluded with an open-ended item (additional 
comments). iJ^st ructions occurring immediately after the item 
requesting occupational status of the subject requested 
individuals not teaching to remit the instrument w^t^hoiit 
responding to the Likert items. Figure 1 presents a portion of 
the items used in this inquiry. The Likert items were organized 
into seven content domains consistent with the skills deemed 
necessary by the 1984 Texas Administrative Code and Statutory 
Cications, Title 19, Part II (Texas Education Agency, 1984) that 
is, Instructional methods, strategies, media, technology (12 
items); Classroom management (6 items); Measurement and 
evaluation (5 items); Needs of special populations (4 items); 
Curriculum planning: Scope and sequence of essential elements as 
specified in the state curriculum document (6 items); Legal and 
ethical aspects of teaching (4 items); Organization and 
management of schools (6 items). An alpha coefficient of 
internal consistency was determined for the 43 Likert items to be 
cC = . 9 3 . The Likert items sought perceptions about teacher 
preparation curricula in general, not the program the former 
student had experienced. 

The instrument, printed on a single sheet of blue 110 lb 
index card stock, was folded in such a manner so the return 
address and postage label appeared on an external surface. The 
courier print font was used for the instrument with black ink. 
Subjects completing the questionnaire simply stapled or taped the 
folded instr nt and remitted it by mail. Recommendations of 
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Sudman and Bradburn (1982) regarding mail survey instruments were 
incorporated into the instrument's format. 
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Data Collection 

Subjects for this inquiry were identified from graduation 
announcements beginning with the Spring 1980 commencement list 
and continued through the Fall 1984 commencement list. Subjects 
were randomly selected with parameters established by the year of 
graduation and number of graduates. 

Addresses for these individuals were obtained from one of 
the following sources: departmental newsletter mailing list, 
records of inactive students held by the department and college, 
and the address file of the Association of Former Students. The 
departmental newsletter mailing list represented the most current 
source, since addresses in this list were updated following each 
newsletter mailout during 1985. Addresses obtained from the 
inactive student records were dated, reflecting the graduates' 
permanent address during their final semester in residence at the 
university. Similarly, the address list form the Association of 
Former Students was last updated in 1983. Concern for the 
accuracy of addresses drawn from these latter sources was well 
founded^ since 38 questionnaires were returned undelivered. 

Because the influence of incentives on the response ratio 
was of interest in this inquiry, 40 subjects from the total 
sample were randomly selected to receive token monetary 
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incentives (quarters) wit\ the initial survey request. Only two 
requests containing the quarters were returned undelivered. It 
was noted that 88 or 33.6 percent of the total sample had 
received one or more departmental newsletters. It was assumed 
the newsletters would serve to foster a stronger association 
between the former students and their Alma riater thus encouraging 
a response to this request for information. 

The initial mailout, consisting of a coverletter which 
emphasized the opportunity to influence teacher preparation 
curricula and a questionnaire, was placed in the mail Tuesday, 
April 23, 1985 . A quarter was attached to the cover letter for 
those students selected to receive the monetary incentive. A 
sentence in their cover letters stated that the quarter was 
intended as a small token of appreciation for their assistance. 

A second mailout was conducted three weeks after the 
original mailing (May 16, 1985) which excluded subjects whose 
questionnaires were returned undelivered. Other subjects who had 
not responded by that time were sent a follow-up packet which 
contained a letter explaining the importance of their response to 
program development and another copy of the questionnaire. Data 
received through June 17 were included in findings of this report 
although a closing date was not mentioned in the request. 

It should be noted that we were aware of the potential 
difficulty of achieving a substantial return given the mailing 
date of the surveys and the brief period between the initial and 
follow-up mailings. The literature (Jackson & Schuyler, 1984) 
and our experience with past survey efforts both signaled 
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limitations with our data collection plans. Unfortunately, other 
projects and scheduled efforts delayed the planning and 
implementation of this inquiry until late in the spring semester. 

Findings 

The influence of demographic characteristics of subjects, 
incentives to enhance returns and multiple requests for 
information on perceptions of survey respondents are reported in 
these findings. A summary of the perceptions toward various 
knowledge and skills for a teacher preparation curricula that is, 
the content of the survey, can be obtained from the authors. 

Survey re ;ponses were received from 125 former students or 
48.3% of the 259 who were thought to hove received 
questionnaires. Sixteen subjects who were not teaching heeded 
the instructions on the instrument and returned the survey 
completing only the biographic information. Since 38 undelivered 
requests were returned by the postal service, it is assumed that 
12.8% ( 38/29 7 ) of the addresses were inaccurate . However, it is 
feasible that additional undelivered questionnaires were not 
returned because 7 of the 38 undelivered requests were received 
from the second mailout based on the postmark date. 

The first mailing yielded 101 returns (70 returned 
instruments and 31 undelivered packets), while the second mailing 
resulted in 62 additional returns (55 returned instruments and 7 
undelivered packets). This response ratio (.483) was lower and 
statistically different [X^{4, N=2697) = 30.23r D<.001] than 
previous mail surveys conducted with degree recipients, i.e.; 
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.63/1974, .63/1977, .58/1978, .60/1983 (Rosser & Denton, 1977; 
Denton, 1979; Denton et al . , 1978; Garcia, Armstrong, Dockweiler 
& Wiseman, 1983). (The J982 survey, alluded to earlier, was a 
telephone survey, thus the response ratio for that effort was not 
included in this analysis). The preceding mail surveys were 
conducted during the middle of the academic year (December- 
February) while the present effort was conducted later in the 
spring (April-May). Comparing the response ratios across the 
surveys conducted within the department, it appears that at least 
one contributing factor to the smaller response ratio was timing 
of the project, i.e., late spring (April-May). Thus one 
recommendation, ancillary to the findings associated with the 
questions of this inquiry, is to conauct mail surveys with former 
students during the middle stages of the academic year. 

Another potentially adverse influence on the response ratio 
of this survey was having been asked to respond to previous 
surveys conducted by ':he department. Given the nature of earlier 
efforts, only the 1983 survey could have overlapped with the 
present survey. A chi-square statistic was applied to the 
returns from the current sample, grouped by year of graduation to 
determine whether the following response ratios across years were 
statistically different (.458/1980, .385/1981, .469/1932, 
.500/1983, .548/1984). The resulting chi-square statistic ( 4 , 
N=297 ) = 2.72, £> . 70 was not statistical ly significant indicating 
that for this inquiry, potentially overlapping survey requests 
from the same source was not a limiting factor. 

The initial research question sought to determine the 

11 



Influence of . . • 10 

influence of ideographic characteristics of subjects, i.e., oate 
of graduation and level of work experience, on their responses. 
Responses to the survey items were converted to the following 
numerical values (5 = substantial emphasis, 4 = moderate 
emphasis, 3 = undecided, 2 = minim?.! emphasis, 1 = no emphasis) 
and stored in the university's mainframe computer for subsequent 
analyses by the SPSSX statistical package (SPSSX, 1984). As 
noted in the discussion of the sample, an alpha level of .05 was 
used as the cricerion for statistical significance in this 
inquiry. Descriptive statistics and inferential tests (F ratios) 
of regression models were performed to compare perceptions of 
subjects responding to the instrument. Descriptive analysis 
revealed instances of missing data across the variables, thus the 
total samples for subsequent analyses were expected to vary. 
Given che unequal number of responses categorized by the 
variables, date of graduation (1980 = 20 , 1981 = 13, 1982 = 26 , 
1983 = 21 , 1984 = 29) and level of work experience (kindergarten 
= 16, elementary school = 49, middle or junior high school = 15, 
high school = 11, other = 18) a forward selection regression 
model with one predictor was used to provide inferential tests of 
perceptions across the 43 survey items. Date of graduation 
served as the independent variable for the initial set of 
regression analyses. These procedures were then repeated with 
level of work experience serving as the independent variable. 

None (0 of 43) of the regression tests for the date-of- 
graduation predictor were statistically different at the .05 
alpha level. Conversely, the regression models with level of 
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work experience serving as the predictor yielded statistically 
different tests for 2 of 43 comparisons. That is, level of work 
experience accounted for variation among responses to the 
following pedagogical knowledge/skills: reading strategies for 
content areas [^(j^^ gS) ^ 16.3, p<.001] with means ranging from 
4.7 (teaching elementary school) to 3.5 (teaching high school); 
instructional applications of microcomputers [F., qq. = 6.6, 
£=.01] with means ranging from 4.7 (teaching high school) to 3.9 
(teaching kindergaruen ) . 

Findings presented to address this research question suggest 
that the ideographic variables of subjects, year-of-gradua t ion 
and level-of-work-exper ience , did not appreciably influence 
perce;_^t ions of subjects responding to this survey on curricula 
for teacher education. 

Research questiori 2 asked whether using incentives to 
encourage responding to a survev request influences the response 
ratio and the perceptions of tho.-e who do respond to the survey 
request. As not^j in the data collection section, incentives 
consisted of a newsletter and/or twenty-five cents. Table 1 
provides ^ summary of the response ratios and statistical 
comparisons associated with incentives used in this inquiry. 
Establishing a communication network via newsletters mailed to 
former students enhanced the corresponding response ratio (.588) 
compared to the raiiio of the no incentive condition (.434). The 
application of chi-square to these two conditions yielded a 
statistically significant test, X^(l, N=142) = 6.27, p<.02). 

In contrast to the p^atistical tests incorporating 
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newsletters, the chi-square test for quarters and no incentives 
was not statistically significant, ( ^ ' N=168) = 1.44, £<.2. 
However, an unexpected finding was the lower response ratio 
(•313) for the twenty-five cent incentive compared to the no 
incentive condition. 



Place table 1 about here 



Data analysis associated with the issue of different 
perceptions or response bias due to incentives produced no 
inrtances in 43 comparisons where the response patterns were 
sufficiently different to be statistically significant. 
Descriptive statistics and a forward selection regression model 
with the incentive condition serving as the independent variable 
were performed to yield these findings. 

The final research question, questio 3, asked whether 
perceptions of subjects who responded to an initial survey 
request differed from perceptions of subjects who responded to a 
second request. Two of 43 comparisons were found to be 
statistically different when the responses were categorized by 
whether the return occurred in conjunction with the first or 
second request for survey information from former students* 
These comparisons were made with the same procedures that were 
used for the previous research question. Different responses 
occurred for the pedagogical knowledge/skills, reinforcement and 
praise t£(i^l09) ~ 5.1, £=.03], and leadership styles of 
administrators [I(i^io9) ^•l' £=^.02]. Greater program emphasis 
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(higher scores) wore recorded for reinforcement and praise from 
responses to the initial survey request (means: 4.7-first 
mailout, 4.3-seccnd mailout). However, for leadership styles of 
aaministrators , this phenomenon was reversed with gre^.cer program 
emphasis occurring for responses to the second survey request 
(means: 3.3-first mailout, 3.9-second mailout). An answer to 
research question three based on these findings is that 
perceptions of subjects were not markedly different whether they 
responded to an initial or second request for information on a 
mail survey. 

Discussion 

Educational decision makers often rely on mail surveys to 
gather information because this technique is adaptable to a 
population with wide geographic dispersion, and is relatively low 
cost especially when compared with personal interviews. An 
additional positive attribute is that mail surveys can be 
conducted in a relatively brief time interval. Yet attempts to 
use the mail questionnaire are often frustrating, because of the 
difficulty in motivating sufficient participation, especially 
from former students. 

This inquiry was conducted to examine potential relatio.ij 
among ideographic characteristics of subjects, incentives to 
enhance returns, and multiple requests for information with the 
perceptions of respondents. Characteristics of subjects included 
year of graduation and level of teaching experience. These 
characteristics were used as categorical variables for a series 
of analyses associated with research question 1. The resulting 
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analyses yield^j two statistically difterent item response 
patterns across 80 comparisons. Th'j low percentage (2.3%) of 
statistically different items resulting from these analyses is 
less than the alpha level (.05) established for thir^ inquiry. In 
other words, the statistically different items in these analyses 
could have occurred due to chance. Thus, findings from this 
inquiry indicate the date of graduation and level of work 
experience did not influence the perceptions of subjects 
regarding topics necessary for teacher preparation curricula. 
Generalizing these findings to the survey research literature 
must be done with caution because these findings are inherently 
linked to the content of the questionnaire. And although the 
content of the questionnaire i3 rather commonplace in teacher 
education, it certainly does not generalize to all mail survey 
instruments . 

Research luestion 2 addressed whether incentives enhanced 
the return of survey instruments and whether incentives 
influenced recorded perceptions. The findings of this inquiry 
were mixed regarding the use of incentives to increase response 
ratios. Individuals who had received newsletters from the 
department prior to receiving the questionnaire responded more 
often than those who had not received the departmental 
newsletter. Perhaps the mechanism at work in this case is 
salience, with those subjects who had received the departmental 
newsletter being aware of program development efforts through 
items in the newsletter, and thus understood the significance of 
their resporises to the program development effort. This 
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explanation is consistent with the importance Heberlein and 
Baumgartner (1978) place on salience in motivating individuals to 
respond to mail surveys. 

In contrast, the use of small monetary incentives (25 cents) 
in this inquiry resulted in a response ratio less than that of 
the no incentive condition. This finding is contrary to the work 
of Armstrong (1975) and Linsky (1975) who have reported that 
monetary incenti^/es consistently enhance response ratios. 
Moreover research of Huck and Gleason (1974) who successfully 
used 25 cent incentives to increase mail survey responses, and 
successful efforts by Furse ei c:l . (1981) and Zusman and Duby 
(1984) with 50 cent and one dollar incentives provide sufficient 
evidence that something wcs amiss in the present study. 
Evidently the 25 cent incentive was not viewed favorably by the 
majority of individuals receiving it in their packets. Perhaps 
inflation has devalued the quarter to the extent it is no longer 
considered a "reward" or token of appreciation as it was a decade 
ago. 

On a more positive note, however, was the related finding 
from this inquiry that response incentives did not influence item 
response patterns. This finding is consistent with the work of 
Nederhof (1983), and Zusman and Duby (1984) who report no 
response bias due to the use of incentives to increase 
participation in mail surveys. 

The final research question of this inquiry sought an 
empirically supported response to the concern whether differences 
in perception to the content of the questionnaire would occur 
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TABl£ 1 

Influence of Incentives on Response Ratios of Subjects flailed Questionnaires 





Incentive 
Condition 


Nuraoer of 
Responses 


Nurber of ^ 
Instruments Mailed 


Response 


Chib 
square 


P 


1. 


No Incentives 


59 


136 


.434 






2. 


Newsletter 


50 


85 


.588 


6.39 


<.02 


3. 


Quarter 


10 


32 


.313 


1.44 


N.S. 


4. 


Quarter & 


6 


6 


1.000 


6.27 


<.02 




Newsletter 














Total 


125 


259 


.483 







a Number adjusted due to the return of undelivered instruments. 

b 2x2 Contigency table (designated reward condition compared with no 
incentives) 
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Skill/Knowledge Program Emphasis 

My rating of the emphasis that should be 
placed on the following content/skills in 
education is: 

substantial moderate undecided minimal no 
emphasis emphasis emphasis eaphasis 

DOT IAIN: INSTRUCTIONAL 
f4ETH0DG, STRATEGIES 
flEDIA, TECHNOLOGY 

1. Time management 

(academic engaged 

time of learners) 

2. Learner motivation 



3. Role of prerequi- 
sites (reviewing pre- 
requisite learning) 

4. Intended learning out- 
comes (objectives) 

5. Learner guidance (cues 
to aid learning) 

6. Learner performance 
(application of new 
learning) 

7. Feedback (information 
on appropriateness of 
performance) 



Figijre 1 

Sample of Likert Items on Survey Instmnent 
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